1. Technical Field
Processing and projection or display of color images on surfaces, on televisions, on game displays, on computers or by other electronic display media. In particular, methods and systems for display power savings, and extending battery life in color image display devices by managing color output while not degrading color image quality.
2. Description of Related Art
The projection and/or display of color images is an active area of commercial research and development. New image display, television, games, computers and projection products and viewing experiences are being launched in the marketplace on a regular basis. In one aspect of the marketplace, digital cinema or video projector technology that utilizes colored light emitting diodes (LEDs) as the source of the primary colors for imaging, offers the promise of extreme, wide color gamut along with very long life, low heat illumination. LED brightness is currently limited, however, requiring three optical systems and three image modulators, i.e., one for each of the red, green, and blue (RGB) color channels, for the brightest images. Current projector lamp technology is of higher brightness and can take advantage of single optical systems and single image modulators using complex color filter wheels to provide full color display. In a second aspect of the marketplace, televisions, game displays and computer displays such as liquid crystal displays (LCDs) are now being introduced with LEDs as the backlit light source to again take advantage of the extreme, wide color gamut, long life and low heat output of LEDs. In a third aspect of the marketplace, projectors, televisions, game displays and computer displays are being introduced with more than the typical three (RGB) colors to improve brightness and expand the color gamut. Such products offer the promise and technical challenge of how to best use the wide color gamut.
In a color image projector, in order to gain the advantage of the available wide color gamut, longer life, and lower heat of LED illumination, and to achieve maximum brightness with a single optical system and single image modulator, the multiple RGB channels may be combined for some portion of time during image frames to produce a white exposure during a pixel, or a portion of an image pixel area can include a clear filter area that transmits all of the LCD backlight spectrum for a 4th white sub-pixel. Adding these multiple RGB channels during an image frame duty cycle or area will increase the brightness, but will also reduce the colorfulness by desaturating the pure RGB colors.
Furthermore, in prior art projectors, color rendering is accomplished by processing each of the RGB channels independently with matrix operators or with one-dimensional color look-up tables. In some projectors, the RGB colors and the combinations of two and three colors may be independently controlled. However, such control does not provide full three-dimensional color processing. With these limited processing options, it is not possible to display images optimally in human visual system (HVS) perceptual terms. For example, it is not possible to render visual lightness contrast without affecting either or both of hue and chroma. Achieving optimal visual processing that provides the brightest, most colorful images, while preserving perceived color accuracy requires three-dimensional color processing.
In providing any color image for viewing by a human observer, whether it is an image printed on a substrate, an electronic display, television, or a projection onto a viewing surface, the perception of color stimuli by the human observer is dependent upon a number of factors. In the International Lighting Vocabulary published in 1987 by the Commission Internationale de l'éclairage (CIE), it is noted as follows: “Perceived color depends upon the spectral distribution of the color stimulus, on the size, shape, structure, and surround of the stimulus area, on the state of adaptation of the observer's visual system, and on the observer's experience of the prevailing and similar situations of observations.”
Moreover, in a treatise on the stained glassed windows at the cathedral at Chartres, The Radiance of Chartres: Studies in the Early Stained Glass of the Cathedral, (Columbia University Studies in Art History and Archaeology, No. 4), Random House, 1st Ed., 1965, author James Rosser Johnson wrote that, “ . . . the experience of seeing these windows . . . is a very complicated experience . . . that spans many aspects of perception.” Yet fundamentally, “ . . . when the spectator enters the Cathedral from the bright sunlight, . . . the visitor must step with caution until his eyes have made a partial dark adaptation . . . then the details of the interior will seem lighter and clearer while, at the same time, the [stained-glass] windows become richer and more intense.”
Adaptation plays a powerful role in the instance depicted in Johnson's narrative. By adapting to the darkness or lower, perceived diffuse white of the cathedral's interior, the colors of the windows appear exceedingly brilliant, invoking a perception, in the words of Vincent Scully, Architecture, The Natural and Manmade, St. Martin's Press, 1991, that, “ . . . transcend[s] the statics of the building masses, the realities of this world . . . [creating] a world of illusion, shaped by and for the heavenly light of the enormous stained glass windows.” While such a perceptual experience is certainly complex and affected by the many characteristics of the human visual system (HVS), the richness of it is largely and simply made possible by the broad extent of sensitivity of the HVS and its innate ability to adapt to its surround.
The HVS is capable of adapting to an incredible range of luminance. For example, the HVS may adapt its light sensitivity over a range of about eight orders of magnitude, e.g., from a starlit, moonlit night having a luminance of about 0.0001 candela per square meter (cd/m2) to a brightly lit summer day of about 600 to 10,000 cd/m2. Equally remarkable is that the HVS may accommodate over five orders of magnitude of luminance at any given instant for the perception of complex visual fields that are routinely experienced. This adaptation occurs relative to diffuse white, i.e., an area in the scene that appears white. The perceptions of lightness and chroma are then relative to this white. The higher the brightness of the perceived white, the lower the brightness and chroma of similarly illuminated objects in the scene will appear to the observer; conversely, the lower its brightness, the brighter and more colorful such objects appear.
This means that changing the stimulus that appears white affects the appearance of all other stimuli in the scene. For a display or projection of an image, these powers of adaptation can be harnessed to expand the gamut of the medium in the perceptual sense. For any image display, and particularly single modulation LED displays such as those employing a digital micromirror device (DMD), the projected image can be made to appear brighter by the addition of light from combining RGB colors for some portion of the image frame time. In so doing, the powers of HVS adaptation are exploited to increase the apparent brightness and lightness contrast of the displayed images. For displays illuminated by red, green, and blue LEDs, although the added light reduces the actual display color gamut provided by the “LED primaries,” the R, G, and B primary colors of the LEDs often exceed the current video standards, such as e.g., ITU Radiocommunication Sector (ITU-R) Recommendation BT.709, which is the United States standard for the format of high-definition television and consumer digital media. Thus some colors which are possible to output by the R, G, and B LEDs, or displays with more than three colors and extended color gamut are not available to be encoded in the input color data for display in accordance with such standards. Optimal use of these extended colors requires full three-dimensional color processing and can be further optimized using knowledge of the HVS. Prior attempts to process the current video standards, such as with one-dimensional color processing and color matrices, or without use of HVS models have resulted in unsatisfactory and unrealistic displayed images and high rates of product return by consumers.
Illustrative of some of these attempts, FIGS. 1A-1D are two-dimensional schematic diagrams of various prior art ways for processing input color data to produce output color data for rendering a color image. FIG. 1A shows a color hue/saturation/contrast/brightness method, depicting the global controls that rotate hue, stretch saturation and contrast and raise brightness. All colors are changed with these controls with no way to isolate a given color or color region like flesh tones. Rin/Gin/Win are input HD709 standard colors, and Rout/Gout/Wout are more pure output LED Colors. There are four controls, and if each control is provided with 20 settings for example, there are 80 global choices.
FIG. 1B shows a color matrix method depicting a linear matrix global control that rotates and scales the color axes. All colors are changed globally with no way to isolate local colors like flesh tones. Rin/Gin/Win are input HD709 standard colors, and Rout/Gout/Wout are more pure output LED colors. If a 3×3 matrix is used, there are nine global choices.
FIG. 1C shows a color gamma tables method depicting gamma global controls that independently maps each input color non-linearly to do things such as increase contrast. It can be seen that, e.g., red changes are the same for all green values. The same relationships occur with other combinations of primary colors. Thus gamma controls are global, with no way to locally isolate colors, such as flesh tones. Rin/Gin/Win are input HD709 standard colors, and Rout/Gout/Wout are more pure output LED colors. With three primary colors having 4096 settings, there are 12288 global choices.
FIG. 1D shows a 2D example of an RGBCYMW seven color mapping method. In this simple example of 7-color tetrahedral processing, the RBG/RGW triangles are independently processed using linear interpolation of input/output control values at each vertices. This is a global control, with no way to isolate local colors or regions like flesh tones. Rin/Gin/Win are input HD709 standard colors, and Rout/Gout/Wout are more pure output LED colors. With 14 In/Out colors, there are 14 global choices. Rin/Gin/Win are input HD709 standard colors, and Rout/Gout/Wout are more pure output LED Colors.
Digital Cinema Initiatives, LLC (DCI) is a joint venture of major motion picture studios, which was formed in 2002 to create standards for digital cinema systems, including image capture and projection. The digital color standard adopted by the studios for professional movie releases in the DCI format is 12 bits per primary color, nonlinear CIE XYZ Tristimulus values. This is the first time that a digital standard has been established that is encoded in visual color space and therefore independent of any imaging device. For example, using this standard, the same digital file can be displayed to produce the specified color on a television or a printer. The color gamut of this digital color standard is larger than any possible display.
FIG. 3 is a diagram of color gamuts, including color gamuts of the DCI and HD709 standards, and color gamuts of various media and/or imaging devices. It can be seen that in diagram 400, the color gamuts 406, 408, 410, and 412 of the various imaging devices are substantially larger than the HD709 standard 404. Accordingly, to take full advantage of the color capabilities of these imaging devices 406-412, the color gamut of the HD709 standard must be mapped upwardly, to render the full colors of the larger color gamut, while simultaneously preserving flesh tones and other memory colors, and optimizing the particular device for viewing in a particular environment.
It can also be seen that the large triangular boundary 402 that represents the DCI standard encompasses all of the color gamuts of the media and/or imaging devices, as well as the color gamut of the HD709 standard 404. Accordingly, the digital color standard input color gamut 402 must be contracted or reduced to fit within the color gamut of a physical display such as a television or projector. Truncating or clipping those input digital color values of the DCI standard that lie outside of the color gamut boundary of the display device will cause loss of color saturation and detail and create a visually sub-optimal displayed image. Conventional video processing using one-dimensional color tables and linear matrices will also produce sub-optimal displayed images. Optimal display of these contracted colors requires full three-dimensional color processing and can be further optimized using knowledge of the HVS and the state of visual adaptation in particular viewing environments.
Also, image and video media display products are now being reduced in size. Examples of such products are the new miniature pico-projectors and portable, handheld displays such as iPods® or iPads®. Because of power, heat, and size limitations, these displays generally have reduced color gamuts due to reduced contrast or reduced color saturation. They also are often used in widely differing viewing environments both indoors and outdoors. Improvement of the overall quality of these smaller gamut displays with conventional image and video input is critical to product value. Conventional video processing using one-dimensional color tables and linear matrices will also produce sub-optimal displayed images. Optimal display of these contracted colors requires full three-dimensional color processing and can be further optimized using knowledge of the HVS and the state of visual adaptation in particular viewing environments.
Additionally, the capabilities of HVS adaptation are affected by the viewing environment. In a dark room, higher contrast is needed in a projected or displayed image for an equally perceived viewing experience as compared to a room with normal room lighting or viewing the same image in bright outdoor lighting. Relative to bright outdoor lighting, the HVS adaptation to the dark room and the lower overall image brightness combine to reduce the perceived image contrast. In a brightly lit room, less contrast is needed due to brightness adaptation and more contrast is needed due to viewing flare from room lights illuminating the dark areas of the displayed image.
In image displays, televisions, and/or projectors using high brightness light sources or expanded or reduced color gamuts, there is therefore a need in displaying and/or projecting images to optimize the increase in perceived brightness, contrast, and colorfulness while preserving expected memory colors of the displayed image such as flesh tones. Such an optimization should take into account that not all colors should be adjusted in the same manner and to the same extent. To do so would result in images containing certain details that appear unsatisfactory to a human observer. For example, if a flesh tone of a face in an image is modified in the same manner as a relatively saturated color of another object in the image, the face will be perceived as “pink,” “orange,” or “burnt” by an observer and thus will be perceived as unsatisfactory. There is therefore a need to achieve this optimization while also preserving certain known colors, such as flesh tones, grey tones, named colors (such as commercial “brand” colors), and other “memory” colors in the image. Prior attempts to process the video inputs with one-dimensional color processing and color matrices for such extended brightness, contrast or color gamut displays, have resulted in unsatisfactory and unrealistic displayed images and high rates of product return by consumers.
Current projectors, televisions or displays that attempt to enhance or improve perceived color quality with processing that is in any way different than exact colorimetric color reproduction, do not preserve memory colors in the background. A memory color may be characterized as a localized volume in a color space, as will be described subsequently herein. The algorithms used in current image displays, televisions and projectors cannot uniquely preserve a volume within a three-dimensional color space while changing a different volume within the same three-dimensional color space using one dimensional tables, or matrices, or enhancements which are applied to all colors in the 3D space. For example, in some image projectors, color enhancement is attempted using output color definitions of the seven input colors RGBCMYW (red-green-blue-cyan-magenta-yellow-white). This may allow one to provide a bright white in an image without changing red, for example, but it does not allow one to specify any point or localized volume of a memory color in a 3D color space, which is required to preserve that memory color. As a result, when current image displays, televisions and projectors provide enhanced colors, they do so across the entire color gamut, “enhancing” certain memory colors such as flesh tones such that a typical human observer finds them unsatisfactory and not perceptually optimal. In such image devices, the color enhancement is somewhat arbitrary; it does not preserve memory colors, nor produce a perceived display image that is realistic for a better viewing environment.
More generally, to the best of the applicants' knowledge, no one has implemented the use of three dimensional color tables in 3D color processing to improve image quality for video images, or in 3D color processing for gamut mapping to larger color gamut displays than a particular image standard, or in gamut mapping to smaller color gamut displays than a particular image standard, or in 3D color mapping to displays with secondary color capability and more than three colors that are primary or secondary, using visual models of the human visual system or otherwise. Currently, standard color processing for displays uses one dimensional tables, 3×3 matrices or matrix mathematics that allows output definition of a small number of colors like RGBCYMW.
3D color tables have been implemented for color calibration, but in such circumstances, the tables are small (e.g., 7×7×7). These 3D look-up-tables are used instead of one dimensional tables and 3×3 matrices because the small 3D look-up-tables are generally faster, albeit at the expense of some loss of precision. In any case, significant color improvement or enhancements to deliver color “looks,” or gamut mapping or mapping to displays with secondary or more than three primary colors with such small tables is not possible.
Another problem in certain types of image rendering devices is that the outputs of the primary color light sources are not stable. This is particularly true for image rendering devices that use organic light emitting diodes (OLEDs) as the sources of the primary colors red, green, and blue. A known problem with OLED displays is that the blue OLED typically has had a considerably shorter lifespan than the red and green OLEDs. One measure of OLED life is the decrease of luminance to half the value of original brightness. The luminance of currently available blue OLEDs decreases to half brightness in a much shorter time than the red or green OLEDs. During the operation of an OLED display, this differential color change between the blue OLED and the red and green OLEDs changes the color balance of the display. This change is much more objectionable to a viewer than a decrease in overall brightness of the display.
To the best of the applicants' knowledge, the problem of managing the overall lifespan of OLED displays has not be solved adequately, which has led to significant delays in product introduction in the marketplace. There is therefore a need to provide a solution that manages the overall quality and lifespan of the relative luminances of the red, green and blue OLEDs in a display device.
Another problem in certain types of image rendering devices is that the battery life of the device is not sufficient to satisfy users' needs. As advances have occurred in wireless communication technology, consumers are spending increasing amounts of time using such devices, while also demanding increased quality in the color images that they display. Mobile displays are being used increasingly for all types of entertainment media including long form media such as television and movies. These uses occur in diverse lighting environments, and the usage lifetime is a critical characteristic to consumers. In a typical hand-held device, both the increase in the duty cycle and the demand for maximum perceived image quality of the display have placed a heavy burden on the batteries used to power such devices.
Consumers are often finding that the battery life of a particular device is less than stated in sales literature and/or owner's manuals. The also find that they cannot use their devices as often as they would like (in some cases nearly continuously over the course of a day), without carrying some sort of a charging cord and plug that connects to a 12 volt automobile jack, a 120 VAC outlet, or a USB port. This is a significant annoyance, and device makers recognize the overall problem.
One solution would be to increase the overall size of the batteries in the device. However, this is clearly unsatisfactory, because space within any of these devices is at a premium, and manufacturers are unwilling to allocate additional volume to batteries when it is needed for many other purposes of equal or greater priority.
Another measure to increase battery life is to reduce the power delivered to the image display, which may be a plasma display, a liquid crystal display (LCD), or an organic light emitting diode (OLED) display. This generally results in a reduction in the brightness of the display, which is also unsatisfactory to consumers, particularly when viewing the display in full daylight. Their expectations of image quality are continuously being raised. For example, a recent iPad® product of Apple launched in Mar. 2012 offers “breakthrough technology” in its display, as set forth at http://www.apple.com/ipad/features/: “The Retina display on the new iPad features a 2048-by-1536 resolution, 44 percent greater color saturation, and an astounding 3.1 million pixels—in the same 9.7-inch space. That's four times the number of pixels in iPad 2 and a million more than an HDTV.” In general, the newest, most desirable image displays on the market are typically demanding more battery power, rather than less.
Consumers are unwilling to trade battery life for image quality. There continues to be an increasing need for color display devices that can provide the highest of image quality while also providing sufficiently long battery life to meet consumer demands. Products which can meet this need will have a significant competitive advantage in the marketplace.